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Motivation  for  Design  Review  Improvements  Topic 


•  Design  escapes  continue  to  impact  program  cost,  schedule  and 
mission  performance 

•  Our  design  review  and  development  test  programs  have  failed  to 
identify  issues  early  enough  to  mitigate  program  or  mission  impacts 

•  Detecting  and  correcting  design  defects  early  in  a  product  life  cycle  is 
becoming  increasingly  difficult  as  space  systems  become  more 
complex 

•  In  hindsight,  many  design  escapes  were  deemed  to  be  preventable 

-  Hypothesis:  Late  design  escapes  could  be  an  indicator  of  a  process  gap 

•  Need  to  assess  if  process  changes  are  needed  to  address  any  identified  gaps 

•  Effects  of  late  design  escapes  can  be  impactful  to  a  company  in 
many  ways: 

-  Costly,  damages  reputations,  strains  customer  relations,  embarrassing 

•  E.g.,  RF  cross  talk  in  a  unit  resulted  in  an  18-month  impact  to  the  program 

-  Preventable  with  the  right  set  of  reviewers  at  the  right  time 
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Design  Review  Improvement  Charter 


•  Identify  the  deficiencies  or  weaknesses  in  the  existing  design  review 
process  that  allowed  design  escapes  to  take  place  by  leveraging 
existing  case  studies  and  escapes 

•  Identify  design  review  process  improvements 

•  Survey  and  assess  the  practices  utilized  across  industry  and 
government  agencies  to  prepare  for  and  conduct  design  reviews 

-  Surveyed  team  member  companies  -  49  test  cases 

-  Reviewed  Aerospace  on-orbit  anomaly  data  (Classified)  -  121  test  cases 
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Escape  Analysis  (1  of  2) 


Inadequate 

Requirements 

6% 


Majority  due  to  inadequate  design 
review  (60%) 

•  RF  crosstalk  in  unit* 

-  Other  causes  include: 

•  Inadequate  analysis  (30%) 

-  Gyro  life  test  failure  * 

•  Inadequate  requirements  (6%) 

-  No  coupling  requirement  for 
military  earth  coverage  (MEC) 
signals  to  earth  coverage 
signals  (EC)* 

•  Review  recommendation  not 
performed  (4%) 

-  Power-on  reset  circuit* 

Reviewer  skillset  (72%)  implicated  in 
cause  of  inadequate  reviews 

-  Not  getting  help,  not  the  right  person(s),  not 
raising  issues 

-  Mixed  technology  units  require  multi-discipline 
SME  reviewers 


Design  Review  Escape  Cause 


Review 

Recommendation 
Not  Performed 
4% 


Inadequate  Inadequate 
Analys's  resign  Review^ 
30%  /%  jp% 


Inadequate  Design  Review  Cause 
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Contract  2% 


*  Denotes  escape  would 
have  been  found  by  a 
fully  tested  EM 
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Escape  Analysis  (2  of  2) 


•  21  of  49  escapes  did  not  have  a  fully 
tested  EM  prior  to  CDR 

•  19  of  21  escapes  noted  above 
could  have  been  detected  with  a  fully 
tested  EM 

-  Designers  indicated  that  the  escapes  would 
have  been  found  had  they  utilized  a  fully 
tested  EM  prior  to  CDR 

-  An  EM  provides 

•  Opportunity  to  discover  design  defects 

•  Analytical  model  validation 

•  Requirements  validation 

•  Build  process  validation 

•  Demonstrate  interface  compatibility 

•  Validate  test  and  operation  procedures 


Would  an  EM  have 
Caught  the  Defect? 
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Design  Review  Improvements  Overview 

Development  Processes  of  Seven  Companies  Were  Reviewed 

Strengths: 

•  Contractor  team  members  who  are  familiar  with  their  own  company’s 
Command  Media  stated  that  they  had  a  formalized  development 
process 

•  Reviewers  are  trained  in  the  development  process  (command  media) 
and  what  to  expect  in  the  data  products 

•  Contractor  team  members  have  lessons  learned  databases  as  part  of 
the  development  process  (by  both  the  design  team  and  review  team) 

•  Have  identified  a  best  practice  for  folding  Lessons  Learned  into  the 
Design  Development  Process  Command  Media 


Note:  Development  Process  is  a  general  concept  that  encompass  both  the  actual  design 
process  and  the  design  review 


U.S.  Space  Program 
Mission  Assurance 

•  • 

Improvement 

Workshop 


U.S.  SPACE  PROGRAM  MISSION  ASSURANCE  IMPROVEMENT  WORKSHOP 
LOCKHEED-MARTIN  |  SUNNYVALE,  CA  |  MAY  5  -  7,  2015 


7 


Design  Review  Improvements  Overview 


Weaknesses 

•  The  scope,  criteria,  and  reviewer  guidance  for  conducting  a  design  review  were 
inconsistent  across  industry  with  opportunities  for  improvement 

-  No  reviewer  minimum  experience  for  participating  in  a  review 

-  No  minimum  lead  time  for  reviewer  to  have  material 
•  Is  not  always  specified  by  contract 

-  No  explicit  requirement  for  reviewer  relevant  experience 

-  The  context  of  lower  level  reviews  becomes  lost  as  the  unit  development  process 
matures  and  becomes  overly  summarized 

-  Review  process  tailored  by  prog  ram -driven  constraints  (time,  schedule,  dollars) 
preserves  the  intent  of  any  given  milestone  even  though  the  design  review  is  not  ready 

•  No  effective  command  media  for  mixed  technology  units  (digital,  RF,  analog/ 
power/  ground,  and  FSW  are  all  separate  disciplines,  reviewed  separately) 

•  Action  item  closure  with  originator  approval  not  consistent 

•  No  requirement  for  having  a  fully  tested  EM  before  CDR 
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Findings  Summary 


•  The  key  to  a  successful  product  development  is  the  experience  and 
skills  of  the  development  engineers 

-  The  expectation  that  a  codified  process  can  catch  all  escapes  is 
unreasonable 

-  It  depends  on  both  a  robust  process  and  the  skills  of  those  involved 

-  Just  as  you  cannot  expect  to  review  in  quality-you  cannot  expect  to  “ review ” 
in  a  good  product 


U.S.  Space  Program 
Mission  Assurance 

•  • 

Improvement 

Workshop 


U.S.  SPACE  PROGRAM  MISSION  ASSURANCE  IMPROVEMENT  WORKSHOP 
LOCKHEED-MARTIN  |  SUNNYVALE,  CA  |  MAY  5  -  7,  2015 


9 


Example  of  Recommendations 

Reviewer  Skill  Set: 

•  Ensure  that  the  development  process  defines  the  minimum  relevant  domain 
experience  to  be  a  Lead  or  Senior  Reviewer 
-  There  should  be  recognized  subject  matter  experts  with  the  relevant 
material  under  review 


Design  Changes  for  Obsolescence  or  Application: 

•  Design  changes  due  to  obsolescence  or  revectoring  for  a  new  application 
(how  used)  needs  a  rigorous-heightened  review  supported  by  test 

Immature  or  Incorrect  Data  Products  or  Unknown-Unknowns: 

•  Utilize  a  fully  tested  EM  in  support  of  CDR 

-  Forces  early  discovery  of  defects  while  maturing  data  products 


Inadequate  analysis  in  context  with  the  desired  application: 

•  Ensure  that  the  development  process  provides  for  the  reviewer  to  review 
the  analysis  scope  in  context  to  the  requirements  as  part  of  the  design 
reviewer’s  tasks 
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Design  Review  Improvement  Product  Traceability 


Deliverable  Requested 

Location  Covered  in  Product 

Identify  strengths  and  weaknesses  of  the  current  design 
review  process  at  the  component/unit/box  level  and  below 

Section  4.1:  Current  Design 

Review  Practices 

Recommend  codified  changes  and/or  upgrades  to  the 
process  that  will  effectively  and  efficiently  identify  and/or 
prevent  design  errors  early  in  the  program  lifecycle 

Section  3.1:  Recommended 

Design  Review  Changes 

Recommend  updates  to  the  entrance  and  exit  criteria  for 
the  design  review  process 

Section  3.1 .1 :  Entry  and  Exit 

Criteria 

Recommend  criteria  for  the  selection  of  independent 
design  reviewers  with  the  proper  subject  matter  expertise 

Section  3.1.2:  Design  Reviewer 
Selection 

Define  the  level  of  technical  rigor  required  to  successfully 
prepare  for  and  conduct  a  thorough  design  review 

Section  3.1.3:  Technical  Rigor 

Define  the  actions  to  be  taken  when  deemed  not  ready  to 
proceed  with  the  design  review 

Section  3.1.4:  Reviewer/Lead 
Responsibilities 

Identify  programmatic  benefits  for  conducting  a  thorough 
technical  design  review 

Section  4.3  Program  Benefits 

U.S.  Space  Program 

Mission  Assurance 

Improvement 

Workshop 
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Intended  Product  Use 


What  is  the  intended  use  of  the  product? 

-  Who  is  the  target  audience? 

•  Engineers,  program  office,  mission  assurance  professionals,  designers,  systems 
engineering,  suppliers,  subcontract  management,  customers,  and  senior  leadership 

-  How  should/could  it  be  used? 

•  The  recommendations  should  be  used  to  augment  the  current  design  review 
process  in  order  to  reduce  escapes,  costs,  and  schedule 

Specific  recommendations  for  industry: 

-  What  should  industry  do  with  the  product  near  term/long  term? 

•  Consider  adopting  recommendations  to  contractor  process 

•  Collect  and  develop  best  practices  for  development  and  review  of  mixed 
technology  units 

Specific  recommendations  for  government: 

-  What  should  government  do  with  the  product  near  term/long  term? 

•  Consider  adopting  recommendations  for  government  participants  in  review  process 

•  Understand  risk  areas  for  escapes  to  better  understand  trades  (e.g.,  EMs) 
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Keith.Henderlong.ffrdc@mda.mil 

Paul  Hopkins 

Lockheed  Martin 

paul .  c .  hopkins  @lmc  o .  c  om 

Daniel  Hyatt 

MDA 

Daniel.Hyatt@mda.mil 

Rolf  Kich 

FMC 

kich@flightmw.com 

C.  J.  Fand 

Harris 

cland@harris .  com 

Ed  Jopson 

Northrop  Grumman 
edward.jopson@ngc.com 

Mark  King 

Micropac 

markking@micropac  .com 

Jim  Farosa 

BAE  Systems 

j  ame  s .  laro  sa@baesy  stems  .com 

Frederick  Kelso 

MDA 

Frederick.Kelso@mda.mil 

Brian  Kosinski 

SSF 

Brian.Kosinski@sslmda.com 

Neil  Fimpanukom 

SSF 

Neil.Fimpanukom@sslmda.com 

Jeanne  Kerr 

Fockheed  Martin 

J  eanne .  R.  Kerr  @lmco .  c  om 

John  Kowalchik 

Fockheed  Martin 

J  ohn.  J .  Ko  walchik@lmco  .com 

Fouie  Fombardo 

Fockheed  Martin 

louie .  lombardo@lmco  .com 

Kurt  Ketola 

Raytheon 

ketola@raytheon.com 

Debbie  Schreiber 

Fockheed  Martin 
debbie.schreiber@lmco.com 

Rob  Fyon 

SSF 

robert.lyon@sslmda.com 

Ronand  Mandel 
Lockheed  Martin 
ronald.h.mandel@lmco.com 


Patrick  Martin 
NASA 

partick.martin@nasa.gov 


Steven  McNeil 
Xilinx 

stevem@xilinx.com 


Kevin  Meadows 
Northrop  Grumman 
kevin.meadows@ngc.com 


Eli  Minson 
Ball 

eminson@ball .  com 


Tom  Musselman 
Boeing 

thomas.e.musselman@boeing.com 


Helen  Narciso 
Lockheed  Martin 
helen.narciso@lmco.com 


John  Nelson 
Lockheed  Martin 
john.d.nelson@lmco.com 


David  Newton 
Northrop  Grumman 
david.a.newton@ngc.com 


Ethan  Nguyen 
Raytheon 

ethan_m_nguyen@raytheon.com 


Jeff  Oberst 
Lockheed  Martin 
j  eff.  ober  st@lmco .  com 


Frank  Pastizzo 
SSL 

frank.pastizzo@sslmda.com 


Kevin  Paxton 
Boeing 

kevin.r.paxton@boeing.com 


David  Pinkley 
Ball 

dpinkley  @ball .  com 


Rob  Pollard 
Ball 

rpollard@ball .  com 


Anne  Ramsey 
Harris 

aramsey  @harris .  com 


Brian  Reilly 
DCMA 

Brian.Reilly@dmca.mil 


Robert  Ricco 
Northrop  Grumman 
bob.ricco@ngc.com 


John  Robinson 

Aerojet  Rocketdyne 

john.robinson@rocket.com 


Frank  Roller 
Lockheed  Martin 
frank.d.roller@lmco.com 


Joseph  Roubal 

Aeroflex,  a  Cobham  Company 
joseph.roubal@aeroflex.com 


Rabindra  Singh 
SSL 

Rob.Singh@sslmda.com 


Melanie  Sloane 
Lockheed  Martin 
melanie .  sloane@lmco  .com 


Homer  D.  Stevens 
Lockheed  Martin 
homer.d.stevens@lmco.com 


Norman  Strampach 

Lockheed  Martin 

norman.  strampach@lmco .  com 


David  Swanson 
Orbital  ATK 

David.Swanson@orbitalatk.com 


Alfred  Tadros 
SSL 

Alfred.  T  adros@sslmda.  com 


Jeffrey  Tate 
Raytheon 

j  effrey_tate@raytheon.  com 


Paul  Thompson 
Intelsat 

PaulAlex.Thompson@intelsat.com 


Brett  Tobey 
Lockheed  Martin 
brett.f.tobey@lmco.com 


Mike  Tolmasoff 
Boeing 

mike.w.tolmasoff@boeing.com 


Ghislain  Turgeon 
SSL 

Ghislain.Turgeon@sslmda.com 


Deborah  Valley 

MIT  Lincoln  Labs 

deborah.valley@ll.mit.edu 


Richard  Veres 
Honeywell 

richard.veres@honewell.com 


Brynn  Watson 
Lockheed  Martin 
brynn.a.watson@lmco.com 


Lance  Werthman 
Lockheed  Martin 
lance.werthman@lmco.com 


Dan  Yasukawa 
Lockheed  Martin 
dan.yasukawa@lmco.com 


Thomas  Fitzgerald 
SMC 

thomas .  fitzgerald.  5  @us .  as .  mil 


Hal  Bell 
NASA 

harold.m.bell@nasa.gov 


Mike  Wadzinski 
MDA 

mike.wadzinski@mda.mil 


Brig  Gen  Anthony  Cotton 
NRO 

cottonan@nro .  mil 


Barry  Liu 
Boeing 

barry .  liu@boeing  .com 


Mick  Worcester 
Boeing 

michael .  s .  worce  ster@boeing  .com 


Cindy  Kohlmiller 
Northrop  Grumman 
cindy.kohlmiller@ngc.com 


Melanie  Berg 
NASA 

melanie.d.berg@nasa.gov 


Derek  Layne 
Northrop  Grumman 
derek.  layne@ngc  .com 


Jeff  Cusick 
Raytheon 

j  eff.  cusick@raytheon.  com 


Luis  Garcia 
Raytheon 

lfgarcia@raytheon.com 


Dyane  Peters 
Raytheon 

dyane_peters@raytheon.com 


Donna  Potter 
SSL 

donna.potter@sslmda.com 
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(AF  OFFICE) 

